{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "412068d6",
   "metadata": {},
   "source": [
    "# Vector Search with Qdrant\n",
    "\n",
    "## Vector Search\n",
    "\n",
    "Vector search is the backbone of the modern internet, whether you notice it or not. It powers recommendation engines, chatbots, AI agents, and even major search engines.\n",
    "\n",
    "In simple terms, traditional keyword search works by matching exact words. This works well when you know the precise keywords present in the data. But what happens when there are no keywords? What if you're searching through images, audio, video or code, or even cross-modally?\n",
    "\n",
    "Moreover, even in text-heavy documents, keyword search struggles to capture context and meaning. The same idea can be phrased in countless ways, so it is completely unfeasible to compare/search for using keyword-based methods.\n",
    "\n",
    "Instead of relying on exact matches, vector search retrieves information based on semantic similarity measured numerically between vectorized data representations (embeddings). It recognizes patterns and relationships between concepts, enabling search systems to retrieve the most relevant content, even when the phrasing differs, terminology varies, or no explicit keywords exist.\n",
    "\n",
    "## Qdrant\n",
    "\n",
    "[Qdrant](https://qdrant.tech/) is an **open-source** vector search engine, a dedicated solution built in Rust for scalable vector search. \n",
    "If you're wondering why you might need a dedicated solution for vector search, we’ve addressed that in the article [\"Built for Vector Search\"](https://qdrant.tech/articles/dedicated-vector-search/).\n",
    "\n",
    "To TLDR:\n",
    "- To make production-level vector search at scale;\n",
    "- To stay in sync with the latest trends and best practices;\n",
    "- To fully use vector search capabilities (including those beyond simple similarity search).\n",
    "\n",
    "In this notebook, we’ll give you a small sneak peek into semantic (vector) search with Qdrant and encourage you to play around & see if it fits your needs!\n",
    "\n",
    "If you have any questions about vector search in Qdrant, feel free to reach out in our [Discord community](https://discord.gg/G7PQU6Cy).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e3d2706",
   "metadata": {},
   "source": [
    "## Step 0: Setup\n",
    "\n",
    "Qdrant is fully open-source, which means you can run it in multiple ways depending on your needs.  \n",
    "You can self-host it on your own infrastructure, deploy it on Kubernetes, or run it in managed Cloud.  \n",
    "\n",
    "We're going to run a Qdrant instance in a Docker container.\n",
    "\n",
    "### Docker\n",
    "\n",
    "All you need to do is pull the image and start the container using the following commands:\n",
    "\n",
    "```bash\n",
    "docker pull qdrant/qdrant\n",
    "\n",
    "docker run -p 6333:6333 -p 6334:6334 \\\n",
    "   -v \"$(pwd)/qdrant_storage:/qdrant/storage:z\" \\\n",
    "   qdrant/qdrant\n",
    "```\n",
    "\n",
    "The second line in the `docker run` command mounts local storage to keep your data persistent.\n",
    "So even if you restart or delete the container, your data will still be stored locally.\n",
    "\n",
    "- 6333 – REST API port\n",
    "- 6334 – gRPC API port\n",
    "\n",
    "To help you explore your data visually, Qdrant provides a built-in **Web UI**, available in both Qdrant Cloud and local instances.\n",
    "You can use it to inspect collections, check system health, and even run simple queries.\n",
    "\n",
    "When you're running Qdrant in Docker, the Web UI is available at http://localhost:6333/dashboard\n",
    "\n",
    "### Installing Required Libraries\n",
    "\n",
    "In the environment you created specifically for this course, we’ll install:\n",
    "\n",
    "- The `qdrant-client` package. We'll be using the Python client, but Qdrant also offers official clients for JavaScript/TypeScript, Go, and Rust, so you can choose the best fit for your own projects.\n",
    "\n",
    "- The `fastembed` package - an optimized embedding (data vectorization) solution designed specifically for Qdrant. Make sure you install version `>= 1.14.2` to use the **local inference** with Qdrant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "0c30d39b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.1.1\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!python -m pip install -q \"qdrant-client[fastembed]>=1.14.2\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52904ad3",
   "metadata": {},
   "source": [
    "## Step 1: Import Required Libraries & Connect to Qdrant\n",
    "\n",
    "Now let’s import the necessary modules from the `qdrant-client` package.\n",
    "\n",
    "The `QdrantClient` class allows us to establish a connection to the Qdrant service,  \n",
    "while the `models` module provides definitions for various configurations and parameters we’ll use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "2caceaae",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/evgeniya_sukhodolskaya/Desktop/Qdrant/Course/venv/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "from qdrant_client import QdrantClient, models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75ab9d9d",
   "metadata": {},
   "source": [
    "Initialize the client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "527eeefc",
   "metadata": {},
   "outputs": [],
   "source": [
    "client = QdrantClient(\"http://localhost:6333\") #connecting to local Qdrant instance"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1eaebe42",
   "metadata": {},
   "source": [
    "## Step 2: Study the Dataset\n",
    "\n",
    "To build a working vector search solution (and, more generally, to understand if/when/how it’s needed), it's good to study the dataset and figure out the nature and structure of the data we’re working with, for example:\n",
    "\n",
    "- modality — is it text, images, videos, a combination?  \n",
    "- specifics — if it’s text: language used, how big are the text pieces, are there any special characters, etc.  \n",
    "\n",
    "It will help us define:\n",
    "- the right data \"schema\" (what to vectorize, what to store as metadata, etc);  \n",
    "- the right embedding model (the best fit based on the domain, precision & resource requirements). \n",
    "\n",
    "We have a toy dataset provided for experimentation, let's check it out:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "af03241a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "docs_url = 'https://github.com/alexeygrigorev/llm-rag-workshop/raw/main/notebooks/documents.json'\n",
    "docs_response = requests.get(docs_url)\n",
    "documents_raw = docs_response.json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c750e127-a082-4d56-826e-7ff2cb53388f",
   "metadata": {},
   "outputs": [],
   "source": [
    "documents_raw"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ee1420a",
   "metadata": {},
   "source": [
    "Data already seems cleaned and chunked (i.e., divided into small pieces that embedding models can easily digest), so what's left is to define:\n",
    "\n",
    "- which fields could be used for **semantic search** ;\n",
    "- which fields should be stored as **metadata**, e.g. useable for filtering conditions; \n",
    "\n",
    "We have a dataset with three course types:  \n",
    "`data-engineering-zoomcamp`, `machine-learning-zoomcamp`, and `mlops-zoomcamp`.  \n",
    "Each course includes a collection of `question` and `text` (answer) pairs, along with the `section` the question refers to.\n",
    "\n",
    "---\n",
    "\n",
    "#### Which Fields Could Be Used for Semantic Search\n",
    "\n",
    "Here we can observe semantic similarity in practice: some of the questions and answers don’t share many overlapping words,  \n",
    "yet they clearly address the same topic. One asks about a topic, the other provides an answer.\n",
    "\n",
    "For example:\n",
    "\n",
    "**Question**:  \n",
    "- *“I have **registered** for the Data Engineering Bootcamp. When can I expect to receive the confirmation email?”*\n",
    "\n",
    "**Answer**:  \n",
    "- *“You don't need it. You're accepted. You can also just start learning and submitting homework without registering. It is not checked against any **registered** list. Registration is just to gauge interest before the start date.”*\n",
    "\n",
    "These two could be matched via the keyword **registered**,  \n",
    "but a sentence like *“Not **registered** participants are not getting certification”* would also match that keyword, while having a different semantic meaning.\n",
    "\n",
    "So, if we’re building a Q&A retrieval-augmented generation (RAG) system,  \n",
    "it makes sense to store the `text` field (answers) as **embeddings**, and use vector search to find the most relevant answer to a given `question` query.\n",
    "\n",
    "#### Which Fields Should Be Stored as Metadata\n",
    "\n",
    "For example, we could store the `course` and `section` fields as metadata.  \n",
    "This way, we can filter search results when asking questions related to a specific course or a specific section.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5fb849bc",
   "metadata": {},
   "source": [
    "## Step 3: Choosing the Embedding Model with FastEmbed\n",
    "\n",
    "Now that we know we're embedding small chunks of English text (course-related question and answer pairs), we can choose a suitable embedding model to convert this data into vectors.\n",
    "\n",
    "The choice of an embedding model depends on many factors:\n",
    "- The task, data modality, and data specifics;\n",
    "- The trade-off between search precision and resource usage (larger embeddings require more storage and memory);\n",
    "- The cost of inference (especially if you're using a third-party provider);\n",
    "- etc\n",
    "\n",
    "> The best way to select an embedding model is to **test and benchmark different options on your own data**.\n",
    "\n",
    "In this notebook, we’re going to use [FastEmbed](https://github.com/qdrant/fastembed) as our embedding provider.\n",
    "\n",
    "---\n",
    "\n",
    "**FastEmbed** is an optimized embedding solution designed specifically for Qdrant. It delivers low-latency, CPU-friendly embedding generation, eliminating the need for heavy frameworks like PyTorch or TensorFlow. It uses quantized model weights and ONNX Runtime, making it significantly faster than traditional Sentence Transformers on CPU while maintaining competitive accuracy.\n",
    "\n",
    "FastEmbed supports:\n",
    "- **Dense embeddings** for text and images (*the most common type in vector search, ones we're going to use today*)\n",
    "- **Sparse embeddings** (e.g., BM25 and sparse neural embeddings)  \n",
    "- **Multivector embeddings** (e.g., ColPali and ColBERT, late interaction models)  \n",
    "- **Rerankers**\n",
    "\n",
    "All of these can be directly used in Qdrant (as **Qdrant supports dense, sparse & multivectors along with hybrid search**).  \n",
    "FastEmbed’s integration with Qdrant allows you to directly pass text or images to the Qdrant client for embedding.\n",
    "\n",
    "In this notebook, we’ll use FastEmbed for local inference with Qdrant.  \n",
    "> Keep in mind your machine's resources when choosing an embedding model for local inference.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e459b7a",
   "metadata": {},
   "source": [
    "### FastEmbed for Textual Data\n",
    "\n",
    "Let’s select an embedding model to use for our course question answers, stored in `text` fields, from the options supported by FastEmbed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b7fd264-1f9a-439d-a26d-9072bb1a412c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastembed import TextEmbedding\n",
    "TextEmbedding.list_supported_models()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a2893f2",
   "metadata": {},
   "source": [
    "It makes sense to choose a model that produces small-to-moderate-sized embeddings (e.g., 512 dimensions), so we don’t overuse resources in our simple setup.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "18dd0905",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"model\": \"BAAI/bge-small-zh-v1.5\",\n",
      "  \"sources\": {\n",
      "    \"hf\": \"Qdrant/bge-small-zh-v1.5\",\n",
      "    \"url\": \"https://storage.googleapis.com/qdrant-fastembed/fast-bge-small-zh-v1.5.tar.gz\",\n",
      "    \"_deprecated_tar_struct\": true\n",
      "  },\n",
      "  \"model_file\": \"model_optimized.onnx\",\n",
      "  \"description\": \"Text embeddings, Unimodal (text), Chinese, 512 input tokens truncation, Prefixes for queries/documents: not so necessary, 2023 year.\",\n",
      "  \"license\": \"mit\",\n",
      "  \"size_in_GB\": 0.09,\n",
      "  \"additional_files\": [],\n",
      "  \"dim\": 512,\n",
      "  \"tasks\": {}\n",
      "}\n",
      "{\n",
      "  \"model\": \"Qdrant/clip-ViT-B-32-text\",\n",
      "  \"sources\": {\n",
      "    \"hf\": \"Qdrant/clip-ViT-B-32-text\",\n",
      "    \"url\": null,\n",
      "    \"_deprecated_tar_struct\": false\n",
      "  },\n",
      "  \"model_file\": \"model.onnx\",\n",
      "  \"description\": \"Text embeddings, Multimodal (text&image), English, 77 input tokens truncation, Prefixes for queries/documents: not necessary, 2021 year\",\n",
      "  \"license\": \"mit\",\n",
      "  \"size_in_GB\": 0.25,\n",
      "  \"additional_files\": [],\n",
      "  \"dim\": 512,\n",
      "  \"tasks\": {}\n",
      "}\n",
      "{\n",
      "  \"model\": \"jinaai/jina-embeddings-v2-small-en\",\n",
      "  \"sources\": {\n",
      "    \"hf\": \"xenova/jina-embeddings-v2-small-en\",\n",
      "    \"url\": null,\n",
      "    \"_deprecated_tar_struct\": false\n",
      "  },\n",
      "  \"model_file\": \"onnx/model.onnx\",\n",
      "  \"description\": \"Text embeddings, Unimodal (text), English, 8192 input tokens truncation, Prefixes for queries/documents: not necessary, 2023 year.\",\n",
      "  \"license\": \"apache-2.0\",\n",
      "  \"size_in_GB\": 0.12,\n",
      "  \"additional_files\": [],\n",
      "  \"dim\": 512,\n",
      "  \"tasks\": {}\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "EMBEDDING_DIMENSIONALITY = 512\n",
    "\n",
    "for model in TextEmbedding.list_supported_models():\n",
    "    if model[\"dim\"] == EMBEDDING_DIMENSIONALITY:\n",
    "        print(json.dumps(model, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47e08302",
   "metadata": {},
   "source": [
    "We need an embedding model suitable for **English text**. \n",
    "\n",
    "It also makes sense to select a **unimodal** model, since we’re not including images in our search, and specifically tailored solutions are usually better than universal ones.\n",
    "\n",
    "It seems like `jina-embedding-small-en` is a good choice!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "08c2fc80",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_handle = \"jinaai/jina-embeddings-v2-small-en\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d7ca508",
   "metadata": {},
   "source": [
    "Like most dense embedding models, `jina-embedding-small-en` was trained to measure semantic closeness using **cosine similarity**.  \n",
    "You can find this information, for example, on the model’s [Hugging Face card](https://huggingface.co/jinaai/jina-embeddings-v2-small-en).\n",
    "\n",
    "> The parameters of the chosen embedding model, including the output embedding dimensions and the semantic similarity (distance) metric, are required to configure semantic search in Qdrant.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b360404",
   "metadata": {},
   "source": [
    "Now we’re ready to configure and use Qdrant for semantic search.\n",
    "To fully understand what’s happening, here’s a quick overview of Qdrant’s core terminology:\n",
    "\n",
    "- **Points** are the central entity Qdrant works with.  \n",
    "  A point is a record consisting of an **ID**, a **vector**, and an optional **payload**.\n",
    "- A **collection** is a named set of points (i.e., vectors with optional payloads) that you can search within.  \n",
    "  *Think of it as the container for your vector search solution, a single business problem solved.*\n",
    "\n",
    "> Qdrant supports different types of vectors to enable different modes of data exploration and search (dense, sparse, multivectors, and named vectors).\n",
    "\n",
    "In this example, we’ll use the most common type, **dense vectors**.\n",
    "\n",
    "Embeddings capture the semantic essence of the data, while the **payload** holds structured metadata.  \n",
    "This metadata becomes especially useful when applying filters or sorting during search. **Qdrant's payloads** can hold structured data like booleans, keywords, geo-locations, arrays, and nested objects.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f485934",
   "metadata": {},
   "source": [
    "## Step 4: Create a Collection\n",
    "\n",
    "When creating a [collection](https://qdrant.tech/documentation/concepts/collections/), we need to specify:\n",
    "\n",
    "*   Name: A unique identifier for the collection.\n",
    "*   Vector Configuration:\n",
    "    *   Size: The dimensionality of the vectors.\n",
    "    *   Distance Metric: The method used to measure similarity between vectors.\n",
    "\n",
    "\n",
    "There are additional parameters you can explore in our [documentation](https://qdrant.tech/documentation/concepts/collections/#create-a-collection). Moreover, you can configure other vector types in Qdrant beyond typical dense embeddings (f.e., for hybrid search). However, for this example, the simplest default configuration is sufficient.\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d24ad190",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Define the collection name\n",
    "collection_name = \"zoomcamp-rag\"\n",
    "\n",
    "# Create the collection with specified vector parameters\n",
    "client.create_collection(\n",
    "    collection_name=collection_name,\n",
    "    vectors_config=models.VectorParams(\n",
    "        size=EMBEDDING_DIMENSIONALITY,  # Dimensionality of the vectors\n",
    "        distance=models.Distance.COSINE  # Distance metric for similarity search\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f26b24e4",
   "metadata": {},
   "source": [
    "## Step 5: Create, Embed & Insert Points into the Collection\n",
    "\n",
    "[Points](https://qdrant.tech/documentation/concepts/points/#points) are the core data entities in Qdrant. Each point consists of:\n",
    "\n",
    "1. **ID**. A unique identifier. Qdrant supports both 64-bit unsigned integers and UUIDs.  \n",
    "2. **Vector**. The embedding that represents the data point in vector space.  \n",
    "3. **Payload** *(optional)*. Additional metadata as key-value pairs.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "69014231",
   "metadata": {},
   "outputs": [],
   "source": [
    "points = []\n",
    "id = 0\n",
    "\n",
    "for course in documents_raw:\n",
    "    for doc in course['documents']:\n",
    "\n",
    "        point = models.PointStruct(\n",
    "            id=id,\n",
    "            vector=models.Document(text=doc['text'], model=model_handle), #embed text locally with \"jinaai/jina-embeddings-v2-small-en\" from FastEmbed\n",
    "            payload={\n",
    "                \"text\": doc['text'],\n",
    "                \"section\": doc['section'],\n",
    "                \"course\": course['course']\n",
    "            } #save all needed metadata fields\n",
    "        )\n",
    "        points.append(point)\n",
    "\n",
    "        id += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58b4b874",
   "metadata": {},
   "source": [
    "Now we’re going to embed and upload points to our collection.\n",
    "\n",
    "First, FastEmbed will fetch&download the selected model (path defaults to `os.path.join(tempfile.gettempdir(), \"fastembed_cache\")`), and perform inference directly on your machine.  \n",
    "Then, the generated points will be upserted into the collection, and the vector index will be built.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "523a8c7f",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fetching 5 files: 100%|██████████| 5/5 [00:01<00:00,  2.93it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "UpdateResult(operation_id=0, status=<UpdateStatus.COMPLETED: 'completed'>)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.upsert(\n",
    "    collection_name=collection_name,\n",
    "    points=points\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3665f5f1",
   "metadata": {},
   "source": [
    "The speed of upsert mainly depends on the time spent on local inference.  \n",
    "To speed this up, you could run FastEmbed on GPUs or use a machine with more resources.\n",
    "\n",
    "In addition to basic `upsert`, Qdrant supports **batch upsert** in both column- and record-oriented formats.\n",
    "\n",
    "The Python client offers:\n",
    "- Parallelization  \n",
    "- Retries  \n",
    "- Lazy batching  \n",
    "\n",
    "These can be configured via parameters in the `upload_collection` and `upload_points` functions.  \n",
    "For details, check the [documentation](https://qdrant.tech/documentation/concepts/points/#upload-points).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d4de735",
   "metadata": {},
   "source": [
    "### Study Data Visually\n",
    "\n",
    "Let’s explore the uploaded data in the Qdrant Web UI at [http://localhost:6333/dashboard](http://localhost:6333/dashboard) to study semantic similarity visually.\n",
    "\n",
    "For example, using the `Visualize` tab in the `zoomcamp-rag` collection, we can view all answers to the course questions (948 points) and see how they group together by meaning, additionally coloured by the course type.  \n",
    "\n",
    "To do that, run the following command:\n",
    "\n",
    "```json\n",
    "{\n",
    "  \"limit\": 948,\n",
    "  \"color_by\": {\n",
    "    \"payload\": \"course\"\n",
    "  }\n",
    "}\n",
    "```\n",
    "\n",
    "This 2D representation is the result of dimensionality reduction applied to `jina-embeddings`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ae7bc33",
   "metadata": {},
   "source": [
    "## Step 6: Running a Similarity Search\n",
    "\n",
    "Now, let’s find the most similar `text` vector in Qdrant to a given query embedding - the most relevant answer to a given question.\n",
    "\n",
    "### How Similarity Search Works\n",
    "\n",
    "1. Qdrant compares the query vector to stored vectors (based on a vector index) using the distance metric defined when creating the collection.\n",
    "\n",
    "2. The closest matches are returned, ranked by similarity.\n",
    "\n",
    "> Vector index is built for **approximate** nearest neighbor (ANN) search, making large-scale vector search feasible.\n",
    "\n",
    "If you'd like to dive into our choice of vector index for vector search, check our article [\"What is a vector database\"](https://qdrant.tech/articles/what-is-a-vector-database/), or, for a more technical deep dive, our article on [Filterable Hierarchical Navigable Small World](https://qdrant.tech/articles/filtrable-hnsw/).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a52a2d2",
   "metadata": {},
   "source": [
    "Let's define a search function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "145dbefb",
   "metadata": {},
   "outputs": [],
   "source": [
    "def search(query, limit=1):\n",
    "\n",
    "    results = client.query_points(\n",
    "        collection_name=collection_name,\n",
    "        query=models.Document( #embed the query text locally with \"jinaai/jina-embeddings-v2-small-en\"\n",
    "            text=query,\n",
    "            model=model_handle \n",
    "        ),\n",
    "        limit=limit, # top closest matches\n",
    "        with_payload=True #to get metadata in the results\n",
    "    )\n",
    "\n",
    "    return results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae38c1e1",
   "metadata": {},
   "source": [
    "Now let’s pick a random question from the course data.  \n",
    "As you remember, we didn’t upload the questions to Qdrant.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "0e6048ba",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"text\": \"If you uploaded manually the fvh 2019 parquet files manually after downloading from https://d37ci6vzurychx.cloudfront.net/trip-data/fhv_tripdata_2019-*.parquet you may face errors regarding date types while loading the data in a landing table (say fhv_tripdata). Try to create an the external table with the schema defines as following and load each month in a loop.\\n-----Correct load with schema defination----will not throw error----------------------\\nCREATE OR REPLACE EXTERNAL TABLE `dw-bigquery-week-3.trips_data_all.external_tlc_fhv_trips_2019` (\\ndispatching_base_num STRING,\\npickup_datetime TIMESTAMP,\\ndropoff_datetime TIMESTAMP,\\nPUlocationID FLOAT64,\\nDOlocationID FLOAT64,\\nSR_Flag FLOAT64,\\nAffiliated_base_number STRING\\n)\\nOPTIONS (\\nformat = 'PARQUET',\\nuris = ['gs://project id/fhv_2019_8.parquet']\\n);\\nCan Also USE  uris = ['gs://project id/fhv_2019_*.parquet'] (THIS WILL remove the need for the loop and can be done for all month in single RUN )\\n\\u2013 THANKYOU FOR THIS \\u2013\",\n",
      "  \"section\": \"Module 4: analytics engineering with dbt\",\n",
      "  \"question\": \"Invalid data types after Ingesting FHV data through parquet files: Could not parse SR_Flag as Float64,Couldn\\u2019t parse datetime column as timestamp,couldn\\u2019t handle NULL values in PULocationID,DOLocationID\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "\n",
    "course = random.choice(documents_raw)\n",
    "course_piece = random.choice(course['documents'])\n",
    "print(json.dumps(course_piece, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb6770f4",
   "metadata": {},
   "source": [
    "Let's see which answer we get:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "612f0b9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "result = search(course_piece['question'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "371b5b53",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "QueryResponse(points=[ScoredPoint(id=310, version=0, score=0.8827988, payload={'text': \"If you uploaded manually the fvh 2019 parquet files manually after downloading from https://d37ci6vzurychx.cloudfront.net/trip-data/fhv_tripdata_2019-*.parquet you may face errors regarding date types while loading the data in a landing table (say fhv_tripdata). Try to create an the external table with the schema defines as following and load each month in a loop.\\n-----Correct load with schema defination----will not throw error----------------------\\nCREATE OR REPLACE EXTERNAL TABLE `dw-bigquery-week-3.trips_data_all.external_tlc_fhv_trips_2019` (\\ndispatching_base_num STRING,\\npickup_datetime TIMESTAMP,\\ndropoff_datetime TIMESTAMP,\\nPUlocationID FLOAT64,\\nDOlocationID FLOAT64,\\nSR_Flag FLOAT64,\\nAffiliated_base_number STRING\\n)\\nOPTIONS (\\nformat = 'PARQUET',\\nuris = ['gs://project id/fhv_2019_8.parquet']\\n);\\nCan Also USE  uris = ['gs://project id/fhv_2019_*.parquet'] (THIS WILL remove the need for the loop and can be done for all month in single RUN )\\n– THANKYOU FOR THIS –\", 'section': 'Module 4: analytics engineering with dbt', 'course': 'data-engineering-zoomcamp'}, vector=None, shard_key=None, order_value=None)])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "561077bb",
   "metadata": {},
   "source": [
    "`score` – the cosine similarity between the `question` and `text` embeddings.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4992645e",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "Let’s compare the original and retrieved answers for our randomly selected question.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "40ca2226",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question:\n",
      "Invalid data types after Ingesting FHV data through parquet files: Could not parse SR_Flag as Float64,Couldn’t parse datetime column as timestamp,couldn’t handle NULL values in PULocationID,DOLocationID\n",
      "\n",
      "Top Retrieved Answer:\n",
      "If you uploaded manually the fvh 2019 parquet files manually after downloading from https://d37ci6vzurychx.cloudfront.net/trip-data/fhv_tripdata_2019-*.parquet you may face errors regarding date types while loading the data in a landing table (say fhv_tripdata). Try to create an the external table with the schema defines as following and load each month in a loop.\n",
      "-----Correct load with schema defination----will not throw error----------------------\n",
      "CREATE OR REPLACE EXTERNAL TABLE `dw-bigquery-week-3.trips_data_all.external_tlc_fhv_trips_2019` (\n",
      "dispatching_base_num STRING,\n",
      "pickup_datetime TIMESTAMP,\n",
      "dropoff_datetime TIMESTAMP,\n",
      "PUlocationID FLOAT64,\n",
      "DOlocationID FLOAT64,\n",
      "SR_Flag FLOAT64,\n",
      "Affiliated_base_number STRING\n",
      ")\n",
      "OPTIONS (\n",
      "format = 'PARQUET',\n",
      "uris = ['gs://project id/fhv_2019_8.parquet']\n",
      ");\n",
      "Can Also USE  uris = ['gs://project id/fhv_2019_*.parquet'] (THIS WILL remove the need for the loop and can be done for all month in single RUN )\n",
      "– THANKYOU FOR THIS –\n",
      "\n",
      "Original Answer:\n",
      "If you uploaded manually the fvh 2019 parquet files manually after downloading from https://d37ci6vzurychx.cloudfront.net/trip-data/fhv_tripdata_2019-*.parquet you may face errors regarding date types while loading the data in a landing table (say fhv_tripdata). Try to create an the external table with the schema defines as following and load each month in a loop.\n",
      "-----Correct load with schema defination----will not throw error----------------------\n",
      "CREATE OR REPLACE EXTERNAL TABLE `dw-bigquery-week-3.trips_data_all.external_tlc_fhv_trips_2019` (\n",
      "dispatching_base_num STRING,\n",
      "pickup_datetime TIMESTAMP,\n",
      "dropoff_datetime TIMESTAMP,\n",
      "PUlocationID FLOAT64,\n",
      "DOlocationID FLOAT64,\n",
      "SR_Flag FLOAT64,\n",
      "Affiliated_base_number STRING\n",
      ")\n",
      "OPTIONS (\n",
      "format = 'PARQUET',\n",
      "uris = ['gs://project id/fhv_2019_8.parquet']\n",
      ");\n",
      "Can Also USE  uris = ['gs://project id/fhv_2019_*.parquet'] (THIS WILL remove the need for the loop and can be done for all month in single RUN )\n",
      "– THANKYOU FOR THIS –\n"
     ]
    }
   ],
   "source": [
    "print(f\"Question:\\n{course_piece['question']}\\n\")\n",
    "print(\"Top Retrieved Answer:\\n{}\\n\".format(result.points[0].payload['text']))\n",
    "print(\"Original Answer:\\n{}\".format(course_piece['text']))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4cbe8fc",
   "metadata": {},
   "source": [
    "Now let’s search the answer to a question that wasn’t in the initial dataset.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "78a51171",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No, late submissions are not allowed. But if the form is still not closed and it’s after the due date, you can still submit the homework. confirm your submission by the date-timestamp on the Course page.y\n",
      "Older news:[source1] [source2]\n"
     ]
    }
   ],
   "source": [
    "print(search(\"What if I submit homeworks late?\").points[0].payload['text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c2b45a9",
   "metadata": {},
   "source": [
    "## Step 7: Running a Similarity Search with Filters\n",
    "\n",
    "We can refine our search using metadata filters. \n",
    "\n",
    "> Qdrant’s custom vector index implementation, Filterable HNSW, allows for precise and scalable vector search with filtering conditions.\n",
    "\n",
    "For example, we can search for an answer to a question related to a specific course from the three available in the dataset.  \n",
    "Using a `must` filter ensures that all specified conditions are met for a data point to be included in the search results.\n",
    "\n",
    "> Qdrant also supports other filter types such as `should`, `must_not`, `range`, and more. For a full overview, check our [Filtering Guide](https://qdrant.tech/articles/vector-search-filtering/)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89878180",
   "metadata": {},
   "source": [
    "To enable efficient filtering, we need to turn on [indexing of payload fields](https://qdrant.tech/documentation/concepts/indexing/#payload-index).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "d75d9f65",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "UpdateResult(operation_id=2, status=<UpdateStatus.COMPLETED: 'completed'>)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.create_payload_index(\n",
    "    collection_name=collection_name,\n",
    "    field_name=\"course\",\n",
    "    field_schema=\"keyword\" # exact matching on string metadata fields\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f03ff71e",
   "metadata": {},
   "source": [
    "Now let's update our search function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "64c3284a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def search_in_course(query, course=\"mlops-zoomcamp\", limit=1):\n",
    "\n",
    "    results = client.query_points(\n",
    "        collection_name=collection_name,\n",
    "        query=models.Document( #embed the query text locally with \"jinaai/jina-embeddings-v2-small-en\"\n",
    "            text=query,\n",
    "            model=model_handle\n",
    "        ),\n",
    "        query_filter=models.Filter( # filter by course name\n",
    "            must=[\n",
    "                models.FieldCondition(\n",
    "                    key=\"course\",\n",
    "                    match=models.MatchValue(value=course)\n",
    "                )\n",
    "            ]\n",
    "        ),\n",
    "        limit=limit, # top closest matches\n",
    "        with_payload=True #to get metadata in the results\n",
    "    )\n",
    "\n",
    "    return results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bad60d7b",
   "metadata": {},
   "source": [
    "Let’s see how the same question is answered across different courses:  \n",
    "`data-engineering-zoomcamp`, `machine-learning-zoomcamp`, and `mlops-zoomcamp`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "80a988ee",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Please choose the closest one to your answer. Also do not post your answer in the course slack channel.\n"
     ]
    }
   ],
   "source": [
    "print(search_in_course(\"What if I submit homeworks late?\", \"mlops-zoomcamp\").points[0].payload['text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5e5dabd",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "🎉 Congratulations! You now have everything you need to run a simple semantic search with Qdrant! 👏\n",
    "\n",
    "In general, data preparation, organization, and storage in a production-ready vector search solution is a topic worth a course of its own.  \n",
    "If you’re curious to dive deeper into efficient vector search setup, check out our [Vector Search Manuals](https://qdrant.tech/articles/vector-search-manuals/).\n",
    "\n",
    "In the next videos, we will show you how to use **[hybrid search](https://qdrant.tech/articles/hybrid-search/)**, combining the strengths of both keywords-based search and vector search. In many real-world applications, they work hand-in-hand, balancing the precision of keywords with the flexibility of embeddings to deliver the best results.\n",
    "\n",
    "P.S. We encourage you to check out Qdrant’s capabilities, which go beyond similarity search powering RAG & agentic pipelines (but still, here's our [MCP server](https://github.com/qdrant/mcp-server-qdrant) ;) ).\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
